Evaluation Metrics:
How Good is Your Model, Really?

The problem with accuracy

Accuracy is defined as the fraction of correct predictions out of all predictions. Simple, intuitive, easy to explain. It is also deeply misleading in many of the situations where you most need a reliable metric.

Consider a fraud detection system. In a typical dataset, maybe 0.5% of transactions are fraudulent. A model that predicts "not fraud" for every single transaction achieves 99.5% accuracy. It catches zero fraudulent transactions. Nobody would use it. Yet accuracy says it is excellent.

The moment your classes are imbalanced, which they almost always are in real fraud, medical, and security applications, you need better tools. This lesson gives you those tools.

Classification metrics that actually matter

Everything you need starts from the confusion matrix. You met it in Lesson 3.2. Now let's build the key metrics from its four cells: True Positives (TP), True Negatives (TN), False Positives (FP), and False Negatives (FN).

The precision-recall tradeoff

Precision and recall are locked in a tug-of-war. If you make your model more aggressive about flagging positives, recall goes up (you catch more real cases) but precision goes down (you also raise more false alarms). If you make it more conservative, precision goes up but recall falls.

Which end of that tradeoff you prefer depends entirely on the cost of each type of error in your specific context.

When you cannot clearly prioritise one over the other, use F1. When your problem has a heavy class imbalance, consider also reporting the balanced accuracy or the Matthews Correlation Coefficient, which account for the imbalance directly.

The ROC curve and AUC-ROC

Most classifiers do not just output a class label. They output a probability. "This email has a 91% chance of being spam." You then apply a threshold: anything above 0.5 gets labelled spam. But that threshold is a choice. Different thresholds give different precision-recall tradeoffs.

The ROC (Receiver Operating Characteristic) curve visualises the full range of this tradeoff across every possible threshold. It plots True Positive Rate (recall) against False Positive Rate (1 minus specificity) as the threshold sweeps from 0 to 1. A perfect classifier hugs the top-left corner. A random classifier follows the diagonal.

AUC-ROC is especially useful because it is threshold-independent. You can compare two models on their AUC without committing to a particular decision threshold upfront. Once you have selected the best model by AUC, you then choose the threshold based on your precision-recall priorities for the deployment context.

Regression evaluation metrics

Regression problems have their own set of metrics, because error in continuous outputs cannot be measured as right or wrong. Every prediction has a magnitude of error, and different metrics weight those magnitudes differently.

When you report regression results to a non-technical audience, MAE is your friend. "Our model predicts delivery time with an average error of 8 minutes" is something any stakeholder can understand immediately. RMSE is harder to interpret but better for comparing models with each other because it penalises bad predictions heavily. R-squared gives the most complete picture but can be misleading on non-linear data.

Putting it all together in code

Here is a single Python snippet that generates a full evaluation report for a classification model: accuracy, precision, recall, F1, and the ROC-AUC score. Copy this pattern and use it on every classification project.

from sklearn.datasets import load_breast_cancer
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split
from sklearn.metrics import (
    accuracy_score, precision_score, recall_score,
    f1_score, roc_auc_score, classification_report
)

# Load data and train a model
X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)

model = RandomForestClassifier(n_estimators=100, random_state=42)
model.fit(X_train, y_train)

# Hard predictions (class labels)
preds = model.predict(X_test)

# Soft predictions (probabilities) for AUC
probs = model.predict_proba(X_test)[:, 1]

# Print full report
print("=== Classification Report ===")
print(f"Accuracy:  {accuracy_score(y_test, preds):.4f}")
print(f"Precision: {precision_score(y_test, preds):.4f}")
print(f"Recall:    {recall_score(y_test, preds):.4f}")
print(f"F1 Score:  {f1_score(y_test, preds):.4f}")
print(f"AUC-ROC:   {roc_auc_score(y_test, probs):.4f}")
print()
print(classification_report(y_test, preds, target_names=['Malignant', 'Benign']))

The AUC-ROC of 0.9955 is exceptional. Even if we lower the decision threshold to catch more malignant cases at the cost of some false alarms, this model will almost certainly outperform any reasonable baseline. The F1 scores for both classes are above 0.95, confirming the model is not just winning because one class dominates the data.